Message propagation-based stereo image matching system

ABSTRACT

A stereo image matching system includes an image processing unit for converting input images inputted from a first and a second image acquisition unit into digital signals to output first and second pixel data; and an image matching unit for computing at least two of an upward, a downward, a forward and a backward message of each pixel by using data values of the first and the second pixel data that are located on a same epipolar line to calculate a disparity value of each pixel by using the computed messages corresponding to adjacent pixel(s). The stereo image matching system employs a parallel pipeline VLSI configuration with a time complexity of O(N). Thus, a plurality of image lines are used for matching, so that correct distance image information is obtained regardless of the conditions of the surrounding environment.

TECHNICAL FIELD

The present invention relates to a message propagation-based stereoimage matching system and, more particularly, to a messagepropagation-based real-time parallel stereo matching system, which usesa multi-line or full image line processing method.

BACKGROUND ART

As well known to those skilled in the art, stereo matching is a methodof reconstructing three-dimensional spatial information from a pair oftwo-dimensional images. As shown in FIG. 1, stereo matching is a methodof detecting a left and a right side pixel corresponding to a specificlocation (X, Y, Z) in three-dimensional space from image lines ofepipolar lines on a right and a left side image, respectively. Thedisparity d between a pair of the corresponding pixels is defined by theequation d=x^(r)−x¹.

The disparity conveys distance information, and the gecmetricalcharacteristic referred to as a “depth” can be calculated therefrom.Therefore, if the disparity is calculated from input images in realtime, three-dimensional distance information and shape information in anobservation space can be measured.

The basic concept of the stereo matching method is disclosed in a thesisby Dhond, et al. (Umesh R. Dhond and J. K. Aggarwal, Configuration fromStereo—a review, IEEE Transactions on Systems, Man, and Cybernetics,19(6):553-572, November/December 1989); and the latest stereo matchingalgorithm is summarized well in a thesis by Szeliski (D. Scharstein andR. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame StereoCorrespondence Algorithms. IJCV 47(1/2/3):7-42, April-June 2002).

The stereo matching method includes a technique based on awinner-take-all process using a local cost; and a technique forestablishing a global energy model and performing energy minimization onthe basis thereof. By using the winner-take-all process, the stereomatching can be performed in a single scan line and the processing speedcan be enhanced, but has a very large disparity error. The global energymodel technique typically includes a graph cut technique, a beliefpropagation technique, etc. Both the graph cut technique and the beliefpropagation technique are based on energy minimization and exhibitexcellent results, but require very long processing times. That is, thebelief propagation technique exhibits excellent stereo matching resultshaving small error by using the relation between upper and lower lines.In the belief propagation technique, messages are received from anadjacent processor during each sequence to be computed, and the computedmessages are sent to an adjacent processor. Therefore, if the beliefpropagation technique is implemented as a hardware device to perform ahigh-speed parallel processing, the processing time can be shortened.Further, the belief propagation technique is configured such thatmessages are transferred from a current pixel position to an adjacentpixel position in a forward, a backward, an upward, and a downwarddirection, respectively.

However, although the above-described conventional real-time algorithmscan perform a high-speed processing, they have large disparity matchingerrors. Further, since the conventional belief propagation technique isa sequential software algorithm, it has a small disparity matchingerror, but requires a long processing time.

DISCLOSURE OF INVENTION Technical Problem

It is, therefore, an object of the present invention to provide a stereomatching method employing a new parallel pipelined algorithm that can beeasily implemented in a single chip based on the principle of the beliefpropagation algorithm using a plurality of scan lines, thus enhancingthe adaptiveness to the surrounding environment and the processing speedwhile reducing the disparity noise and the production cost.

Technical Solution

In accordance with an embodiment of the present invention, there isprovided a stereo image matching system, including an image processingunit for converting input images inputted from a first and a secondimage acquisition unit into digital signals to output first and secondpixel data; and an image matching unit for computing at least two of anupward, a downward, a forward and a backward message of each pixel byusing data values of the first and the second pixel data that arelocated on a same epipolar line to calculate a disparity value of eachpixel by using the computed messages corresponding to adjacent pixel(s).

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and features of the present invention willbecome apparent from the following description of embodiments given inconjunction with the accompanying drawings, in which:

FIG. 1 shows a conceptual diagram of stereo image matching;

FIG. 2 presents a block diagram showing a message propagation-basedreal-time parallel stereo matching system using a multi-line or fullimage line processing method in accordance with the present invention;

FIG. 3 offers a block diagram of an image matching unit for performingan algorithm I in accordance with a first embodiment of the presentinvention;

FIG. 4 provides a block diagram of an image matching unit for performingan algorithm II in accordance with a second embodiment of the presentinvention;

FIG. 5 is a block diagram of an image matching unit for performing analgorithm III in accordance with a third embodiment of the presentinvention;

FIG. 6 shows a block diagram of an image matching unit for performing analgorithm IV in accordance with a fourth embodiment of the presentinvention;

FIG. 7 provides a block diagram showing an internal configuration ofeach processor for processing respective lines in the image matchingunit for performing the algorithm IV in accordance with the fourthembodiment of the present invention;

FIG. 8 presents a block diagram showing the internal configuration offorward, backward, upward and downward processors constituting the imagematching unit in accordance with the present invention;

FIG. 9 is a block diagram showing the internal configuration of aforward cost processor in accordance with the present invention;

FIG. 10 offers a block diagram showing the internal configuration of abackward cost processor in accordance with the present invention;

FIG. 11 provides a block diagram showing the internal configuration of amatching cost processor in accordance with the present invention;

FIG. 12 is a block diagram showing the internal configuration of adisparity computation processor in the image matching unit in accordancewith the present invention;

FIG. 13 shows a flow chart of the operating process of the imagematching unit for performing the algorithm I in accordance with thefirst embodiment of the present invention;

FIG. 14 illustrates a flow chart showing the operating process of theimage matching unit for performing the algorithm II in accordance withthe second embodiment of the present invention;

FIG. 15 depicts a flow chart showing the operating process of the imagematching unit for performing the algorithm III in accordance with thethird embodiment of the present invention;

FIG. 16 is a flow chart showing the operating process of the imagematching unit for performing the algorithm IV in accordance with thefourth embodiment of the present invention;

FIG. 17 presents a flow chart of a disparity computation in thealgorithms I to IV in accordance with the present invention;

FIG. 18 shows a flow chart of forward, backward, upward and downwardmessage processing in the algorithms Ito IV in accordance with thepresent invention;

FIG. 19 is a flow chart of the forward cost computation algorithm ofFIG. 18;

FIG. 20 illustrates a flow chart of the backward cost computationalgorithm of FIG. 18;

FIG. 21 depicts a flow chart of a matching cost computation algorithm inaccordance with the present invention; and

FIGS. 22 to 28 are pictures showing output images to compare thedisparity of a conventional image matching method to that of the presentinvention.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described indetail with reference to the accompanying drawings. The presentinvention has a parallel pipeline configuration for computing an upward,a downward, a forward and a backward message of each pixel in order tocompute disparity values for the respective pixels in an image by havingprocessors send and receive the computed messages corresponding toadjacent pixels. Further, the present invention has a configuration canbe implemented by a Very Large-Scale Integration (VLSI) for computationof respective messages.

FIG. 2 is a block diagram showing a message propagation-based real-timeparallel stereo matching system using a multi-line or full image lineprocessing method in accordance with the present invention.

As shown in the drawing, the parallel stereo matching system inaccordance with the present invention includes an image processing unit30 for converting images inputted from first and second cameras 10 and20, which are image acquisition means, into digital signals, and thusoutputting first and second pixel data; and an image matching unit 40for computing specific messages using first and second pixel data on thesame epipolar line, and outputting determined values to a user system 50using the computed messages. In this case, the first and second pixeldata are preferably pixel data existing in right and left image lines ofthe same epipolar line.

The image matching unit 40 varies mainly depending on algorithms I, II,III and IV.

First, symbols that will be used in later description are summarizedbelow.

Hereinafter, t_(f) denotes a forward processing time, t_(b) denotes abackward processing time, t_(u) denotes an upward processing time, t_(d)denotes a downward processing time, t_(o) denotes a disparitycomputation time, and t_(all) denotes the total time required forforward, backward, upward and downward processing and disparitycomputation. The times t_(f), t_(b), t_(u), t_(d), t_(o), and t_(all)can be of a same length when parallel processing is performed.

FIG. 3 is a block diagram of an image matching unit 40 for performing analgorithm I in accordance with a first embodiment of the presentinvention.

Right and left side pixel data g^(r)(j+d(t_(all)),i) and g¹(j,i) arestored in a scan line buffer 4011. Further, a forward processor PE^(f)4012 receives the right and the left side image pixel data, and stores amessage m^(f)(t_(f),j) computed by using the image pixel data values ina stack 4013. When a backward processor PE^(b) 4014 also receives theright and the left side image pixel data g^(r)(j+d(t_(all)),i) andg¹(j,i), and sends a message m^(b)(t_(b)) computed by using the imagepixel data to a disparity computation processor PE^(o) 4015, thedisparity computation processor PE^(o) 4015 receives the forward messagem^(f)(t_(f),j) stored in the stack 4013 from the stack 4013, thebackward message m^(b)(t_(b)) and the right and the left side imagepixel data, and thus outputs a disparity value.

FIG. 4 is a block diagram of an image matching unit 40 for performing analgorithm II in accordance with a second embodiment of the presentinvention.

Right and left side image pixel data g^(r)(j+d(t_(all)),i) and g¹(j,i)are stored in a scan line buffer 4021. A forward processor PE^(f) 4022receives the right and the left side image pixel data from the scan linebuffer 4021, a downward message m^(d)(t_(all),j,i−1) computed for anupper line from a buffer 4026, and stores a forward messagem^(f)(t_(f),j) computed by using the received image pixel data anddownward message in a stack 4023. When a backward processor PE^(b) 4024also receives the right and the left side image pixel datag^(r)(j+d(t_(all)),i) and g¹(j,i) and the downward messagem^(d)(t_(all),j,i−1), and sends a backward message m^(b)(t_(b)) computedby using the image pixel data and the downward message to a disparitycomputation processor PE^(o) 4025, the disparity computation processorPE^(o) 4025 receives the forward message m^(f)(t_(f),j) from the stack4023, the backward message m^(b)(t_(b)), the downward message, and theright and the left side image pixel data, and thus outputs a disparityvalue. The downward processor PE^(d) 4027 receives the downward messagefor the upper line, the backward message, the forward message, and theimage pixel data, computes a downward message m^(d)(t_(d),j,i), andstores the downward message in the buffer 4026.

FIG. 5 is a block diagram of an image matching unit 40 for performing analgorithm III in accordance with a third embodiment of the presentinvention. The algorithm III has an iterative computation structure, inwhich an iteration loop is represented by f.

Right and left side image pixel data g^(r)(j+d(t_(all)),i) and g¹(j,i)are stored in a scan line buffer 4031. Further, a forward processorPE^(all) 4032 receives the right and the left side image pixel datag^(r)(j+d(t_(all)),i) and g¹(j,i) from the scan line buffer 4031, upwardand downward message values m^(u)(t_(all),j,i+1,f−1) andm^(d)(t_(all),j,i−1,f−1) computed respectively for a lower line and anupper line in a previous iteration loop from a buffer 4036, computes aforward message m^(f)(t_(f),j), and stores the forward message value ina stack 4033. When a backward processor PE^(b) 4034 also receives theright and the left side image pixel data and the upward and downwardmessages and sends a backward message m^(b)(t_(b)) computed by using thereceived image pixel data and messages to a disparity computationprocessor PE^(o) 4035, the disparity computation processor PE^(o) 4035receives the forward message stored in the stack 4033, the backwardmessage, the upward message, the downward message, and the right and theleft side image pixel data, and thus outputs a disparity value. Adownward processor PE^(d) 4037 receives the downward message for theupper line, the backward message, the forward message, and the imagepixel data, computes a new downward message m^(d)(t_(d),j,i,f), andstores the downward message in the buffer 4036. An upward processorPE^(u) 4038 receives the upward message for the lower line, the backwardmessage, the forward message, and the image pixel data, computes a newupward message m^(u)(t_(u),j,i,f), and stores the upward message in thebuffer 4036.

FIG. 6 is a block diagram of an image matching unit 40 for performing analgorithm IV in accordance with a fourth embodiment of the presentinvention. The algorithm IV has an iterative computation structure, inwhich an iteration loop is represented by f. Further, p number ofprocessors PE-0, PE 1, PE2, PE2, . . . perform parallel processing forhigh-speed processing.

The right and the left side image pixel data are inputted from cameraimages and stored in a scan line buffer 4041. The processors PE0, PE1,PE2, . . . for P number of respective lines are arranged in a parallelmanner, and simultaneously process the P number of lines and thensimultaneously process subsequent P number of lines if processing hasbeen terminated. Therefore, as shown in FIG. 6, in the P number oflines, a first processor performs first line processing, and a p-thprocessor performs p-th line processing. In this way, a single entireimage is processed only after processing is performed a total of N/Ptimes when the number of image lines is N. Each processor receives rightand left side image pixel data g^(r)(j+d(t_(all)),q*P+p) and g¹(j,q*P+p)from the scan line buffer 4041, a downward messagem^(d)(t_(d),j,q*P+p,f) from an upper buffer 4046 to communicate withadjacent processors, and an upward message m^(u)(t_(u),j,q*P+p,f) from alower buffer 4046, processes the received messages, and stores theupward message in the upper buffer 4046 and the downward message in thelower buffer 4046. All processors perform parallel processing whilesending and receiving messages at high speed in this way.

FIG. 7 is a diagram showing the internal configuration of each of theprocessors PE0, PE1, PE2, . . . for processing respective lines in theimage matching unit for performing the algorithm IV in accordance withthe fourth embodiment of the present invention.

A forward processor PE^(f) 4042 receives right and left side image pixeldata g^(r)(j+d(t_(all)),q*P+p) and g¹(j,q*P+p), an upward message valuem^(u)(t_(all),j,q*P+p+1,f−1) and a downward message valuem^(d)(t_(all),j,q*P+p−1,f−1), which are computed for a lower line and anupper line, respectively, in a previous iteration loop, computes aforward message value m^(f)(t_(f),j), and stores the forward messagevalue in a stack 4043. When a backward processor PE^(b) 4044 alsoreceives the right and the left side image pixel data and the upward anddownward messages, and sends a backward message m^(b)(t_(b)) computed byusing the received image pixel data and messages to a disparitycomputation processor PE^(o) 4045, the disparity computation processorPE^(o) 4045 receives the forward message stored in the stack 4043, thebackward, upward, and downward messages, and the right and the left sideimage pixel data, and thus outputs a disparity value. A downwardprocessor PE^(d) 4047 receives the downward message for the upper line,the backward and forward messages, and the image pixel data, and thuscomputes a new downward message. An upward processor PE^(u) 4048receives the upward message for the lower line, the backward and forwardmessages, and the image pixel data, and thus computes a new upwardmessage.

FIG. 8 is a diagram showing the internal configuration of the forward,backward, upward and downward processors, constituting the imagematching unit 40 in the real-time parallel stereo matching system inaccordance with the present invention.

Each of the processors is configured to compute a message, and includesa matching cost processor 4051 for receiving the right and the left sideimage pixel data g^(r)(j+d(t_(i))) and g¹(j) to compute a matching cost,a forward cost processor 4052 for receiving the matching cost andmultiple message values to compute a forward cost, a stack 4053 forreceiving and storing the forward cost, a backward cost processor 4054for receiving the forward cost from the stack 4053 to compute a backwardcost, and a recursive buffer 4055 for allowing the backward cost to berecursively used for subsequent processing.

FIG. 9 is a diagram showing the internal configuration of the forwardcost processor 4052 in accordance with the present invention. In FIG. 9,the forward cost processor is shown as being divided into two circuitsfor simplicity, but, actually, the forward cost processor is configuredby a single circuit.

The forward cost processor 4052 includes subtractors 40521 forsubtracting input parameters from input messages, respectively, a firstadder 40522 for adding the output values of respective subtractors 40521to matching cost, a comparator 40523 for outputting the lower valuebetween the output value of the first adder 40522 and the output valueof a second adder 40525, a first delay buffer 40524 for delaying theoutput value of the comparator 40523 by 1 clock, the second adder 40525for adding a cost C₀ to the output value of the first delay buffer40524, a second comparator 40526 for comparing the output value of thefirst adder 40522 with another input value and outputting the lowervalue between the two values, a second delay buffer 40527 for delayingthe result value of the second comparator 40526 by 1 clock and providingthe delayed result value to the second comparator 40526 as another inputvalue, and a third adder 40528 for adding the output value of the secondcomparator 40526 to cost K₁ and outputting the added value.

FIG. 10 is a diagram showing the internal configuration of the backwardcost processor 4054 in accordance with the present invention. In FIG.10, the backward cost processor 4054 is shown as being divided into twocircuits for easy depiction, but, actually, the backward cost processoris configured by a single circuit.

The backward cost processor 4054 includes a first comparator 40541 forselecting the lower value between the input cost and the output value ofa first adder 40543, a first delay buffer 40542 for delaying the outputvalue of the first comparator 40541 by 1 clock and providing the delayedoutput value to the first adder 40543, the first adder 40543 for addinga cost value C₀ to the output value of the first delay buffer 40542, asecond comparator 40544 for comparing the output value of the firstcomparator 40541 with an input parameter value, and outputting the lowervalue of the two values, a second adder 40545 for adding the outputvalue of the second comparator 40544 to the output value of a seconddelay buffer 40546, the second delay buffer 40546 for delaying theoutput value of the second adder 40545 by 1 clock and providing thedelayed output value to the second adder 40545 as an input value, and ashifting unit 40547 for shifting the output value of the second adder40545 by a certain amount and outputting the shifted value as aparameter.

FIG. 11 is a diagram showing the internal configuration of the matchingcost processor 4051 in accordance with the present invention.

The matching cost processor 4051 includes an absolute differencecomputation unit 40511 for receiving right and left side image pixeldata to compute the absolute value of the difference between the twoimage pixel data values, and a comparator 40512 for comparing the outputvalue of the absolute difference computation unit 40511 with a parameterK₂ to output the lower value of the two values.

FIG. 12 is a diagram showing the internal configuration of the disparitycomputation processors 4015, 4025, 4035, and 4045 in the image matchingunit 40 in the real-time parallel stereo matching system in accordancewith the present invention.

Each of the computation processors includes subtractors 40051 forsubtracting input parameters from corresponding input messages, amatching cost processor 40052 for computing matching cost by using rightand left side image pixel data, an adder 40053 for adding the outputvalues of respective subtractors 40051 to the output value of thematching cost processor 40052, a delay buffer 40054 for delaying theoutput value of a comparator 40055 by 1 clock, the comparator 40055 foroutputting the lower value between the output value of the delay buffer40054 and the output value of the adder 40053, a T-counter 40056 forindicating the steps of disparity computation processing, and adisparity output buffer 40057 for storing the value of the T-counter40056 when the value input from the adder 40053 is the lower valuebetween the two input values of the comparator 40055.

Hereinafter, the operating sequence of the real-time parallel stereomatching system in accordance with the present invention for respectivecomponents will be described in detail.

FIG. 13 is a flow chart showing the operating process of the imagematching unit 40 for performing the algorithm I in accordance with thefirst embodiment of the present invention.

When the number of image lines is N, and the number of pixels in oneline is M, computation is individually performed on an i-th image lineand a j-th pixel in the line. In the i-th line, the present algorithm isperformed forward on the j-th pixel in the sequence of 0 to M−1 at stepsS111 to S117, and then performed backward on the j-th pixel in thesequence of M−1 to 0 at steps S121 to S128. In forward processing, theforward processor of FIG. 3 receives right and left side image pixeldata values, computes a forward message value at step S114, and storesthe forward message value in a stack at step S115. Further, in backwardprocessing, the backward processor of FIG. 3 receives right and leftside image pixel data values, and computes a backward message value atstep S123, and the disparity computation processor of FIG. 3 reads thebackward message value, and the forward message value from the stack,and computes a disparity value at step S124.

FIG. 14 is a flow chart showing the operating process of the imagematching unit 40 for performing the algorithm II in accordance with thesecond embodiment of the present invention.

Computation is individually performed on an i-th image line and a j-thpixel in the image line. In the i-th image line, the present algorithmis performed forward on the j-th pixel in the sequence of 0 to M−1 atsteps S211 to S217, and is performed backward on the j-th pixel in thesequence of M−1 to 0 at steps S221 to S229. In the forward processing,the forward processor of FIG. 4 receives right and left side image pixeldata values and a downward message value for an upper line, computes aforward message value at step S214, and stores the forward message valuein a stack at step S215. Further, in the backward processing, thebackward processor of FIG. 4 receives the right and the left side imagepixel data values and the downward message value for the upper line, andcomputes a backward message value at step S223. The downward processorof FIG. 4 receives the right and the left side image pixel data values,the forward and backward messages, and the downward message for theupper line, and computes a downward message value for a current line atstep S224. The disparity computation processor of FIG. 4 reads thedownward, backward and forward message values and thus computes adisparity value at step S225.

FIG. 15 is a flow chart showing the operating process of the imagematching unit 40 for performing the algorithm III in accordance with thethird embodiment of the present invention.

Computation is individually performed on an f-th iteration loop, an i-thimage line in the step, and a j-th pixel in the image line. The presentalgorithm is repeatedly performed on the entire image F times, thusreducing noise in a disparity image. In an i-th line, the algorithm isperformed forward on the j-th pixel in the sequence of 0 to M−1 at stepsS311 to S318, and is then performed backward on the j-th pixel in thesequence of M−1 to 0 at steps S321 to S332. In the forward processing,the forward processor of FIG. 5 receives right and left side image pixeldata values, a downward message value for an upper line in an (f−1)-thiteration loop, and an upward message value for a lower line in the(f−1)-th iteration loop, computes a forward message value at step S315,and stores the forward message value in a stack at step S316. Further,in the backward processing, the backward processor of FIG. 5 receivesthe right and the left side image pixel data values, the downwardmessage value for the upper line in the (f−1)-th iteration loop, and theupward message value for the lower line in the (f−1)-th iteration loop,and thus computes a backward message value at step S323. The downwardprocessor of FIG. 5 receives the right and the left side image pixeldata values, the forward and backward message values, and the downwardmessage value for the upper line in the (f−1)-th iteration loop, andthen computes a downward message value for a current line at step S324.The upward processor of FIG. 5 receives the right and the left sideimage pixel data values, the forward and backward message values, andthe upward message value for the lower line in the (f−1)-th iterationloop, and thus computes an upward message value for a current line atstep S324. The disparity computation processor of FIG. 5 reads theupward and downward message values computed in the (f−1)-th iterationloop, and the backward and forward message values, and thus computes adisparity value at step S325.

FIG. 16 is a flow chart showing the operating process of the imagematching unit 40 for performing the algorithm IV in accordance with thefourth embodiment of the present invention.

The algorithm is repeatedly performed on the entire image F times, and Pnumber of processors perform processing in parallel for high-speedprocessing. That is, this algorithm is configured to process thealgorithm III in parallel at a high speed. Processors for respectiveimage lines are arranged in a parallel manner, simultaneously processthe P number of lines and then subsequent P number of lines. Therefore,as shown in FIG. 6, in P lines, a first processor performs first lineprocessing, and a p-th processor performs p-th line processing. In thisway, when the number of image lines in a single image is N, the entireimage is completely processed only if the processing is performed atotal of Q=N/P times.

In an i-th line, the processing is performed forward on the j-th pixelin the sequence of 0 to M−1 at steps S411 to S418, and then backward onthe j-th pixel in the sequence of M−1 to 0 at steps S421 to S431. In theforward processing, the forward processor of FIG. 7 receives right andleft side image pixel data values, a downward message value for an upperline in an (f−1)-th iteration loop, and an upward message value for alower line in the (f−1)-th iteration loop, computes a forward messagevalue at step S415, and stores the forward message value in a stack atstep S416. Further, in the backward processing, the backward processorof FIG. 5 receives the right and the left side image pixel data values,the downward message value for the upper line in the (f−1)-th iterationloop, and the upward message value for the lower line in the (f−1)-thiteration loop, and thus computes a backward message value at step S423.The downward processor of FIG. 7 receives the right and the left sideimage pixel data values, the forward and backward message values, andthe downward message value for the upper line in the (f−1)-th iterationloop, and thus computes a downward message value for a current line atstep S424. The upward processor of FIG. 7 receives the right and theleft side image pixel data values, the forward and backward messagevalues, and the upward message value for the lower line in the (f−1)-thiteration loop, and thus computes an upward message value for a currentline at step S424. The disparity computation processor of FIG. 5 readsthe upward and downward message values in the (f−1)-th iteration loop,and the backward and forward message values, and thus computes adisparity value at step S425.

FIG. 17 is a flow chart showing a process of a disparity computation inthe algorithms I to IV in accordance with the present invention.

At steps S508 and S509, an input message m_(k)(t) is a vector in which thas state values ranging from 0 to D−1, and m_(k)(−1) is an inputparameter. U₀(−1) is initialized as K(U₀(−1)=K, a maximum possiblevalue) at step S501, and right and left side image pixel data values andrespective state values of the input message are inputted in eachiteration loop at step S502. Further, a matching cost is computed byusing the right and the left side image pixel data values by thematching cost processor of FIG. 11 at step S503, and the computedmatching cost is added to a value obtained by subtracting the inputparameter from the input message at step S504. A newly computed value U(t) is compared to the lowest value U₀(t−1) computed in a previousiteration loop at step S505, and the lowest value U₀(t) in a currentiteration loop is computed at steps S506 and S507. Further, when thenewly computed value U_(i)(t) is found to be the lowest value, aniteration loop number t is stored in a d register, and a value d isoutputted after all steps have been terminated at step S510. That is,this algorithm is operated such that the lowest value of U_(i)(t) in alliteration loops is obtained, and a loop number t at that time isoutputted as the value d.

FIG. 18 is a flow chart showing the forward, backward, upward anddownward message processing in the algorithms Ito IV in accordance withthe present invention.

The matching cost computation processor of FIG. 8 receives right andleft side image pixel data at step S601, and computes matching costm_(data)(t) at step S602. The forward cost processor of FIG. 8 receivesthe matching cost m_(data)(t), an input message and m₀(t_(buf)) inputtedfrom the recursive buffer of FIG. 8, computes a forward cost at stepS603, and then stores the forward cost in a stack at step S604. Thebackward cost processor of FIG. 8 loads the forward cost at the stack atstep S605, and outputs the backward cost at steps S606 and S607. Herein,the output backward cost is inputted to the recursive buffer of FIG. 8,and can be recursively used for performing the algorithm next time. Them₀(t_(buf)) is provided as a message input when the forward costprocessor performs recursive computation during a procedure forcomputing forward and downward messages.

FIG. 19 is a flow chart showing a process of the forward costcomputation algorithm of FIG. 18.

At steps S707 to 709, an input cost m_(k)(t) is a vector in which t hasvalues ranging from 0 to D−1, and m_(k)(−1) is an input parameter.U₀(−1) is initialized as K (U₀(−1)=K) at step S701, an input cost valueis inputted in each iteration loop at step S702, and the input costvalue is added to a value obtained by subtracting an input parameterfrom the input cost at step S703. A newly computed value U_(i)(t) iscompared to U₀(t−1)+C₀ computed in a previous iteration loop at stepS704, and the lowest value is computed as U₀(t) in a current iterationloop at steps S705 and S706.

FIG. 20 is a flow chart showing a process of the backward costcomputation algorithm of FIG. 18.

At steps S810 and S811, an input message U_(i)(t) is a vector in which thas state values ranging from −1 to D−1, and m_(k)(−1) is an inputparameter. U₀(−1) is initialized as K(U₀(−1)=K), a maximum possiblevalue) at step S801. In each iteration loop, each state value of theforward cost U_(i)(t) is inputted at step S802. U_(i)(t) is compared tom_(a)(t−1)+C₀ computed in a previous iteration loop, and the lowestvalue is set to be the value m_(a)(t) in a current iteration loop atsteps S803 to S805.

Then, m_(a)(t) is again compared to an input parameter U_(i)(−1), andthe lower value between the two values is computed and outputted asm₀(t) at steps S806 to S808.

After m_(s)(t) is initialized as 0, it is added to m₀(t) in eachiteration loop at step S809, and m_(s)(D−1) is shifted rightward by A₀in a final iteration loop at step S812. The shifted value is anormalization parameter for a message.

FIG. 21 is a flow chart showing a process of a matching cost computationalgorithm in the real-time parallel stereo matching system in accordancewith the present invention.

First, right and left side image pixel data values are received, and theabsolute value a(t) of the difference between the pixel data values iscomputed at step S901, and is compared to K₁ at step S902, and the lowervalue between the two values is computed as m_(data)(t) at steps S903and S904.

FIGS. 22 and 23 illustrate input right and left side images, and FIG. 24illustrates a reference disparity output image. FIG. 25 illustrates thecomputation result of another algorithm using only a single scan linefor the right and the left side images (scan line optimizationalgorithm). FIG. 26 illustrates a disparity output image obtained byperforming the algorithm I in accordance with the first embodiment ofthe present invention, FIG. 27 illustrates a disparity output imageobtained by performing the algorithm II in accordance with the secondembodiment of the present invention, and FIG. 28 illustrates a disparityoutput image obtained by performing the algorithm III in accordance withthe third embodiment of the present invention or the algorithm IV inaccordance with the fourth embodiment of the present invention.

As the algorithm develops from the algorithm I into algorithm II,algorithm III, and then algorithm IV, horizontal stripe noises areeliminated, thus reducing disparity error.

As described above, the present invention can achieve a high-speedprocessing, because it has a parallel pipeline configuration that can beimplemented in the form of VLSI, unlike a conventional sequentialsoftware algorithm. That is, the present invention is advantageous inthat, since a plurality of image lines are used for matching, theadaptiveness to the surrounding environment can be enhanced and thematching error can be reduced. Further, in accordance with the presentinvention, by employing an improved parallel pipeline VLSI configurationwith a time complexity of O(N), the real-time processing can beimplemented effectively.

Further, the present invention is advantageous in that, since aplurality of image lines are used for matching, disparity output imagenoise can be eliminated, and this it is possible to obtain distanceimage information that is correct regardless of the conditions of thesurrounding environment.

While the invention has been shown and described with respect to theembodiments, it will be understood by those skilled in the art thatvarious changes and modifications may be made without departing from thescope of the invention as defined in the following claims.

The invention claimed is:
 1. A stereo image matching system, comprising:an image processing unit for converting input images inputted from afirst and a second image acquisition unit into digital signals to outputfirst and second pixel data; and an image matching unit for computing atleast two of an upward, a downward, a forward and a backward message ofeach pixel by using data values of the first and the second pixel datathat are located on a same epipolar line to calculate a disparity valueof each pixel, and iteratively repeating said computation by exchangingmatching information between adjacent pixels from locations of pixelsadjacent a current location of pixels in a 2D grid structure of an imageto locations of peripheral pixels, and collecting the messagespropagated from peripheral pixels, wherein the image matching unitincludes: a scan line buffer for storing therein the right and the leftside image pixel data; a forward processor for receiving the right andthe left side image pixel data to compute a forward message; a stack forsaving the forward message computed by the forward processor; a backwardprocessor for receiving the right and the left side image pixel data tocompute a backward message; and a disparity computation processor forreceiving the right and the left side image pixel data, the forwardmessage from the stack, and the backward message to compute thedisparity value, and wherein, in order to calculate the message, theforward, backward, upward and downward processors include: a matchingcost processor for receiving the right and the left side image pixeldata to compute a matching cost; a forward cost processor for receivingthe matching cost and multiple message values to compute a forward cost;a stack for receiving and storing therein the forward cost; a backwardcost processor for receiving the forward cost from the stack to computea backward cost; and a recursive buffer for allowing the backward costto be recursively used for subsequent processing.
 2. The stereo imagematching system a claim 1, wherein the image matching unit performsimage matching with respect to every line such that, after a forwardprocessing for computing the forward message in a first horizontaldirection with respect to each line to store the computed forwardmessage in the stack has been completed, a backward processing forcomputing the backward message in a second horizontal direction withrespect to said each line is performed to compute the disparity value.3. The stereo image matching system of claim 1, wherein the forwardprocessor further receives a downward message corresponding to an upperadjacent line to compute forward message and the backward processorfurther receives the downward message corresponding to the upperadjacent line to compute the backward message.
 4. The stereo imagematching system of claim 3, wherein the image matching unit performsimage matching by computing the downward message with respect to theupper adjacent line, computing the forward and the backward message byusing the computed downward message, computing the disparity value, andcomputing the downward message with respect to a current line that is tobe used for a lower adjacent line.
 5. The stereo image matching systemof claim 4, wherein the forward processor is operated during the forwardprocessing, and the backward processor, the downward processor and thedisparity computation processor are operated during the backwardprocessing.
 6. The stereo image matching system of claim 1, wherein theforward processor further receives an adjacent upward messagecorresponding to a lower adjacent line computed in a previous iterationloop and an adjacent downward message corresponding to an upper adjacentline computed in the previous iteration loop to compute the forwardmessage, the backward processor further receives the adjacent upwardmessage corresponding to the lower adjacent line computed in theprevious iteration loop and the adjacent downward message correspondingto the upper adjacent line computed in the previous iteration loop tocompute the backward message, and the disparity computation processorfurther receives the adjacent upward message corresponding to the loweradjacent line computed in the previous iteration loop and the adjacentdownward message corresponding to the upper adjacent line computed inthe previous iteration loop to compute the disparity value, and furthercomprising; a downward processor for receiving the right and the leftside image pixel data, the forward message, the backward message and theadjacent downward message to compute a current downward messagecorresponding to a current line; and an upward processor for receivingthe right and the left side image pixel data, the forward message, thebackward message and the adjacent upward message to compute a currentupward message corresponding to a current line.
 7. The stereo imagematching system of claim 6, wherein the image matching unit performs theimage matching with respect to an entire image repeatedly for a numberof times to reduce a noise in a disparity image.
 8. The stereo imagematching system of claim 6 or 7, wherein the forward processor isoperated during the forward processing, and the backward processor, theupward processor, the downward processor and the disparity computationprocessor are operated during the backward processing.
 9. The stereoimage matching system of claim 1, wherein the image matching unitincludes a parallel processor unit having a plurality of said processorsarranged in parallel; and two or more message buffers for storingtherein the upward message and the downward message, wherein each of theprocessors receives the right and the left side image pixel data fromthe scan tine buffer, the upward message from one of the message buffersthat has the upward message stored therein and the downward message fromanother of the message buffers that has the downward message storedtherein to process each image line, and then said each of the processorsstores the downward message in said one of the message buffers and theupward message in said another of the message buffers.
 10. The stereoimage matching system of claim 9, wherein the image matching systemprocessing a plurality of lines by simultaneously operating theprocessors arranged in parallel.
 11. The stereo image matching system ofclaim 9, wherein the parallel processor unit includes: a forwardprocessor for receiving the right and the left side image pixel data, anadjacent upward message corresponding to a lower adjacent line from anadjacent one of the message buffers and an adjacent downward messagecorresponding to a upper adjacent line from another adjacent one of themessage buffers to compute a forward message; a stack for storingtherein the computed forward message; a backward processor for receivingthe right and the left side image pixel data, the adjacent upwardmessage corresponding to the lower adjacent line from said adjacent oneof the message buffers and the adjacent downward message corresponding:to the upper adjacent line from said another adjacent one of the messagebuffers to compute a backward message; a disparity computation processorfor receiving the right and the left side image pixel data the forwardmessage from the stack, the backward message, the adjacent upwardmessage, and the adjacent downward message to compute a disparity value;a downward processor for receiving the right and the left side imagepixel data, the forward message, the backward message and the adjacentdownward message to compute a current downward message; and an upwardprocessor for receiving the right and the left side image pixel data,the forward message, the backward message, and the adjacent upwardmessage to compute a current upward message.
 12. The stereo imagematching system of claim 9, wherein the image matching unit performsimage matching with respect to an entire image repeatedly for a numberof times, and a number of processors processes a number of linesparallelly, the number of the processors being equal to that of thelines.
 13. The stereo image matching system of claim 9 or 12, whereineach of the processors are operated parallelly, the forward processorbeing operated during forward processing, and the backward processor,the upward processor, the downward processor and the disparitycomputation processor being operated during backward processing.
 14. Thestereo image matching system of claim 1, wherein the forward costprocessor includes: a subtractor for subtracting an input parameter fromthe received multiple message values; a first adder for adding an outputvalue of the subtractor to the matching cost; a first delay buffer fordelaying a first input value by 1 clock; a second adder for adding anoutput value of the first delay buffer to a specific control parameter;a first comparator for outputting a lower value between an output valueof the first adder and an output value of the second adder to the firstdelay buffer as the first input value; a second comparator foroutputting a lower value between the output value of the first adder anda second input value; a second delay buffer for delaying an output valueof the second comparator by 1 clock to provide the delayed value to thesecond comparator as the second input value; and a third adder foradding a control parameter to the output value of the second comparator.15. The stereo image matching system of claim 1, wherein the backwardcost processor includes: a first comparator for selecting a lower valuebetween the inputted forward cost and a comparative input value; a firstdelay buffer for delaying an output value of the first comparator by 1clock; a first adder for adding an output value of the first delaybuffer to a specific control parameter to provide the added value to thefirst comparator as the comparative input value; a second comparator foroutputting a lower value between an output value of the first comparatorand an input parameter; a second adder for adding an output value of thesecond comparator to an additional input value; a second delay bufferfor delaying an output value of the second adder by 1 clock to providethe delayed value to the second adder as the additional input value; anda shifting unit for shifting the output value of the second adder by aspecific amount to output the shifted value as a parameter.
 16. Thestereo image matching system of 1, wherein the matching cost processorincludes: an absolute difference computation unit for receiving theright and the left side image pixel data to compute an absolute value ofa difference between the right and the left side image pixel data; and acomparator for outputting a lower value between an output value of theabsolute difference computation unit and a control parameter,
 17. Thestereo image matching system of 1, wherein the disparity computationprocessor includes: a subtractor for subtracting an input parameter fromthe inputted message values; a matching cost processor for computingmatching cost by using the right and the left side image pixel data anadder for adding an output value of the subtractor to an output value ofthe matching cost processor; a comparator for outputting a lower valuebetween an output value of the adder and a secondary input value; adelay buffer for delaying an output value of the comparator by 1 clockto provide the delayed value to the comparator as the secondary inputvalue; a counter for indicating a step number of each iteration loop ofthe disparity computation processor; and a disparity output buffer forstoring an output value of the counter if the output value of thecomparator is the output value of the adder.
 18. The stereo imagematching system of 1, wherein the disparity computation processorperforms the steps of: receiving the right and the left side image pixeldata and a state value of an input message value in each iteration loop;computing the matching cost; adding the input message value to a valueobtained by subtracting an input parameter from the computed matchingcost; comparing a newly computed value with a previous lowest valuecomputed in a previous iteration loop to compute a current lowest valuein a current iteration loop; storing a step number of the currentiteration loop in a register when the newly computed value is thecurrent lowest value; and outputting the value stored in the registerafter all the iteration loops have been completed.
 19. The stereo imagematching system of 1, wherein the forward and the backward processorperform a forward processing in which the forward cost processorcomputes the forward cost in each clock step by using a cost valuecomputed by the matching cost computation processor, an input messagevalue and a cost value stored in the recursive buffer to store thecomputed forward cost in the stack, and, wherein, after the forwardprocessing has been completed, the forward and the backward processorperform a backward processing in which the backward cost processorcomputes the forward cost from the stack to output a backward cost, andinputs the outputted backward cost to the recursive buffer.
 20. Thestereo image matching system of claim 19, wherein the forward costprocessor performs the steps of initializing the delay buffer by amaximum possible value; reading an input cost in each iteration loop;adding an input message value to a value obtained by subtracting aninput, parameter from the input cost; outputting a lower value betweenthe added value a value obtained by adding a control parameter to aprevious value stored in. the delay buffer, and storing the lower valuein the delay buffer.
 21. The stereo image matching system of claim 19,wherein the backward cost processor performs the steps of initializingthe first delay buffer by a maximum possible value; initializing thesecond delay buffer by 0; reading the forward cost in each iterationloop; comparing the forward cost with a comparative value obtained byadding a control. parameter to a value stored in the first delay buffer;storing in the first delay buffer a lower value between the forward costand the comparative value; outputting a lower value between an outputvalue of the first delay buffer and an input parameter; adding a currentoutput value of the first delay buffer to a previous output value of thefirst delay buffer in each iteration loop to store the added value inthe second delay buffer; and shifting the stored value in the seconddelay buffer by a specific amount to compute and output a parameter.